A spectral preserved model based on spectral contribution and dependence with detail injection for pansharpening

Pansharpening integrates the high spectral content of multispectral (MS) images and the fine spatial information of the corresponding panchromatic (PAN) images to produce a high spectral-spatial resolution image. Traditional pansharpening methods compensate for the spatial lack of the MS image using the PAN image details, which easily causes spectral distortion. To achieve spectral fidelity, a spectral preservation model based on spectral contribution and dempendence with detail injection for pansharpening is proposed. In the proposed model, first, an efficacy coefficient (CE) based on the spatial difference between the MS and PAN images is designed to suppress the impact of the detail injection on the spectra. Second, the spectral contribution and dependence (SCD) between the MS bands and pixels are considered to strengthen the internal adaptation of the spectra. Finally, a spectrally preserved model based on CE and SCD is designed to force the fused image fidelity in spectra when the MS image is pansharpened with the details of the PAN image. Experimental results show that the proposed model is effective.

A spectral preserved model based on spectral contribution and dependence with detail injection for pansharpening Lei  Pansharpening integrates the high spectral content of multispectral (MS) images and the fine spatial information of the corresponding panchromatic (PAN) images to produce a high spectral-spatial resolution image. Traditional pansharpening methods compensate for the spatial lack of the MS image using the PAN image details, which easily causes spectral distortion. To achieve spectral fidelity, a spectral preservation model based on spectral contribution and dempendence with detail injection for pansharpening is proposed. In the proposed model, first, an efficacy coefficient (CE) based on the spatial difference between the MS and PAN images is designed to suppress the impact of the detail injection on the spectra. Second, the spectral contribution and dependence (SCD) between the MS bands and pixels are considered to strengthen the internal adaptation of the spectra. Finally, a spectrally preserved model based on CE and SCD is designed to force the fused image fidelity in spectra when the MS image is pansharpened with the details of the PAN image. Experimental results show that the proposed model is effective.
A high-resolution multispectral (HRMS) image is important for the analysis, planning, utilization and management of earth resources 1 . In fact, it is difficult for most satellites to produce HRMS images due to their physical constraints 2 . However, they use two remote sensors with contradictory functions to visit the earth. One sensor provides multispectral (MS) data with coarse spatial information 3 . Another sensor offers panchromatic (PAN) data that includes fine spatial information without spectral content 4 . To meet the demands of remote sensing data applications, fusing the data derived from the aforementioned two kinds of sensors is effective. Generally, this data processing is called pansharpening 5 .
The various research communities take different paths to expand the pansharpening methods that are primarily divided into three families. One family is the component substitution (CS) cluster, where a transformation is performed to shift the MS data into different domains, and one of them is replaced by a PAN band before the reconstituted components are inversed into the original domain. The common CS methods pansharpen remote sensing data with the intensity-hue-saturation (IHS) transform 6 , principal component analysis (PCA) 7 , and the Gram-Schmidt (GS) method 8 . The fused results generated by the CS methods easily exhibit serious spectral distortion. In contrast, the second family, called multiresolution analysis (MRA), affords an HRMS image with spatial distortion because MRA methods integrate the information of remote sensing data at multiple scales formed by multiple transformations 9 . The popular MRA methods are based on the wavelet transform, such as the nonsubsampled contourlet transform (NSCT) 10 and the nonsubsampled shearlet transform (NSST) 11 . A critical comparison among CS and MRA pansharpening algorithms can be seen in literature 12 .
Recently, the third family, called the detail injection model (DIM), based on a hybrid of CS and MRA, has played an important role in the pansharpening field. The DIM injects the details from the PAN image into the MS image to improve the resolution of the fusion image 13 . Although the reduction of the spectral-spatial distortion is easier to implement than both the CS and MRA methods, spectral heterogeneity easily appears in the fused results because the spatial enhancement influences the spectral fidelity 14 .
To overcome this problem, a spectral preservation model based on spectral contribution and dempendence with detail injection for pansharpening is proposed. A spectral recovery algorithm is designed to construct the spectral preservation model where the spectral properties, including the spectral contribution and dependence (SCD) from the pixels in the MS image and the impact of the injection details on the original spectra, are CE based on the extracted details. We let LRMS k denote the LRMS image. This is shown in Fig. 1.
LRMS k is obtained by resampling and interpolating the MS image. The I component is produced by weighting the average LRMS k . To provide the desired details, we adopt a Gaussian filter 6 with human visual characteristics to filter I to obtain the details M detail because LRMS k has abundant color, but it has a coarse edge and texture. Meanwhile, we use the guided filter 15 with the I component as the guided image to filter the PAN to obtain the details PAN detail , because PAN detail in Eq. (1) needs to be highly correlated to the LRMS k . In this study, the Gaussian filter and guided filter act on I and PAN two times. In the filtering operation, the input image minus the output image elucidates the details. The process of obtaining the details of LRMS k and PAN can be seen as follows: (1) FMS k = LRMS k + g k · * PAN detail , k = 1, . . . , B, www.nature.com/scientificreports/ where I l1 , I l2 , LRPAN 1 and LRPAN 2 with a lower resolution are the approximate versions of I and PAN at the 1 and 2 levels of the filtering operation, respectively, and M detail and PAN detail are the details from LRMS k and PAN , respectively. According Eq. (1), PAN detail should be injected into LRMS k with the help of g k . Thus, the original coarse edge and texture, M detail in LRMS k , is enhanced. However, Zhou et al. 16 thought PAN detail increases the intensity and affects edge restoration of the MS image, but that changes the hue and the saturation of the spectral value of each pixel. When only PAN detail is injected, the spectra of the current or neighborhood pixels closely related to M detail may be influenced. To suppress the impact of the spatial enhancement on the spectra, Zhou et al. 16 proposed an efficacy coefficient (EC) based on the spatial difference of the extracted details M detail and PAN detail to modulate the spectra of LRMS k as follows: where max{·} is a function to define the maximum value and (i, j) is the coordinate of a pixel. The performance of the EC k had been verified by applying it to a model of FMS k = (1 + EC k ). * LRMS k + PAN detail in literature 16 .
In this paper, we introduce EC k into Eq. (1), the Eq. (1) can be converted into the equation as follows: Thus, a prototype of the spectral preservation model is formed.
SCD algorithm based on spectral contribution and dependent. Masi et al. 17 confirmed that the different MS bands contain different spectral components, such as vegetation, water, and soil, and strong energy variations exist in the components associated with the spectral bands. Therefore, we think there is a spectral contribution and dependence to exist not only between the MS pixels but also between the MS bands. To achieve spectral fidelity, the optimization of Eq. (6) is not enough. It is necessary to consider the spectral contribution and dependence between the MS bands and the pixels to strengthen the internal adaptation of the spectra. In our work of literature 14 , we quantize the spectral contribution rate between the MS bands given by where (i, j) is the coordinate of the pixel of row i and column j in an image, SC k is a matrix, SC k (i, j) is the spectral contribution rate of the pixel of row i and column j in the matrix SC k . The performance of the SC k had been proved by applying it to a model of 14 . Subsquently, we model the spectral dependence between the MS pixels by finding the eigenroots of a judgment matrix constructed by pixel pairwise judgment in an MS band. Let x k be a column vector of the pixels of the kth MS band with size M = m × n , which are arranged in lexicographical order. Let x i k be the projection of LRMS k (i, j) . After arranging each band of LRMS image into a vector in lexicographical order, we construct the judgment matrices Z k of the kth corresponding band with the formula as follows: where z k i,j is the element of the coordinate position (i, j) in the matrix Z k . Mathematically, the matrix Z k can be expressed as follows: where M = m × n . We solve Z k to generate the eigenvectors w k . First, the columns of Z k are normalized to obtain Finally, we obtain w k = (c k 1 , c k 2 , ..., c k M ) T and rearrange w k to construct a matrix SD k with size r × c as follows: As a result, the spectral preservation algorithm can be defined as follows: where α p k is a spectral modulation coefficient. The proposed spectral preservation model can be described as follows: A high-quality fusion image can be provided by using the Eq. (12).
FMS k = α p k · * LRMS k + g k · * PAN detail = (1 + EC k . * SC k . * SD k ) · * LRMS k + g k · * PAN detail ,  Fig. 3, the size of the LRMS image is 128 × 128, and the size of the corresponding reference image and PAN image is 512 × 512. Especially, we upsample (4 × 4) and interpolate the LRMS image to the scale of the PAN image before fusion. Two types of quantitative metrics, including with reference and without reference, are shown in Table 1. Eight methods shown in Table 2 are compared with the proposed method.  Correlation Coefficient (CC) 13 QNR is composed of D and D S Universal Image Quality Indices (UIQI) 13 Root Mean Square Error (RMSE) 13 Relative Average Spectral Error (RASE) 13 D is the spectral distortion index Spectral Angle Mapper (SAM) 24 Erreur Relative Global Adimensionnelle De Synthese (ERGAS) 13 The peak signal-to-noise ratio (PSNR) 25 D S is the spatial distortion index Q2n-index (Q4) 25    Tables 3, 4, 5 and 6, where the bold black data denotes the best results. Specifically, to more accurately distinguish the difference of the fused images visually, the residual images shown in Figs. 2, 3 and 4b1-j1 are made by subtracting the reference image and the fused images. Subjectively, there is a substantial feature difference between the various satellite datasets. The images in Figs. 2, 3 and 4have rich color, but the color difference is small and the tone is gentle in the images in Figs 5, 6. Experimental results confirm that our method shows excellent performance, but the compared methods have worse and more unstable performances. For example, from Figs. 2, 3, 4, 5 and 6, the results afforded by the MMMT method are blurry, and the GSA, CBD, BDSD, BF and RBDSD methods fuse the WorldView-2 and IKONOS images to result in various degrees of spectral distortion. Specifically, serious spectral distortion is caused by the GSA, BDSD and RBDSD methods in Fig. 2b,d and g. Although the fusion results of the ASIM, DIM,  www.nature.com/scientificreports/   www.nature.com/scientificreports/ and the proposed methods is difficult to distinguish subjectively, the results from the residual images show that our method has the best performance, and the results from Tables 3, 4, 5 and 6 quantitatively confirm that our method outperforms all the comparison methods because the objective value of our method is the best observing the quantized values of the corresponding Figs. 2, 3, 4, 5 and 6 both with reference and without reference, except for the second in SAM index of the Fig. 3, 4 and the second in UIQI index of Fig. 4 in Tables 3, 4 and 5, and the third in the D index of the Fig. 6 in Table 6.

Conclusion
In this paper, a spectral preservation model is constructed to fuse remote sensing images. The proposed model deals with pansharening with adaptive detail injection, while also enforcing spectral fidelity by implementing a spectral preservation algorithm. The proposed algorithm considers not only the impact of the injection detail on the spectra but also the spectral dependence existing between the MS pixels, and bands to reduce the spectral distortion. Two groups of images with different attributes are used in the reduced-scale and full-scale experiments. Eight compared methods and two types of popular quantitative metrics are employed to test the performance of the proposed model. The results confirm that the proposed model can effectively suppress the influence of spatial enhancement on the spectra and strengthen the internal adaptation of the spectra to ensure spectral fidelity.

Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.